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(54) Video signal processing for electronic watermarking 



(57) An image or sequence of images is water- 
marked without limiting the watermark signal. The wa- 
termarking apparatus includes a conventional DCT unit 
(105) and quantizer (110) for generating an array of 
quantized DCT coefficients. The array is watermarked 
(115) by masking the array to select certain ones of the 
DCT coefficients that are then replaced by zero values 



to form a masked array, the masked array is further 
processed by a watermark inserter(l20) that replaces 
the zero valued coefficients with predefined watermark 
coefficients to form a watermarked array of DCT coeffi- 
cients, e.g. , a watermarked image. A decoder for decod- 
ing the bitstream thusly generated and for removing the 
embedded watermark is also taught. 
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tized DCT coefficients and sets the value of each select- 
ed coefficient to zero A control signal, produced by the 
watermark generator, defines the particular coefficients 
that are to be masked The masked array of coefficients 
containing ;he "zeroed" coefficients is coupled to the 
embedded decoder 116 and used to produce a predict- 
ed image. 

[0017] The masked block of DCT coefficients is cou- 
pled to the watermark inserter 120, where the zeroed 
coefficients are replaced with watermark coefficients. 
The watermark coefficients are supplied by the water- 
mark control signal. 

[0018] The watermarked block of DCT coefficients is 
then coupled to the output processor that combines the 
motion vectors and header information with the DCT co- 
efficients and conventionally encodes the assemblage 
using zig-zag scanning, run-level coding (RLC) and var- 
iable length coding (VLC) to form an MPEG compliant 
bitstream. A control signal is provided to block 1 1 2 such 
certain watermark decoding information can be inserted 
into, for example, the user data fields of the bitstream. 
For example, when using a dynamically variable water- 
mark, the synchronization information is inserted into 
the bitstream to enable a decoder to find and extract the 
watermark. Also, for pseudo randomly generated water- 
marks, a seed for decoding the watermark is inserted 
into the bitstream as, for example, picture layer user da- 
ta. As such, the watermark related decoding information 
can be inserted as often as every picture, but need not 
be. 

[0019] The masked block of coefficients is coupled to 
the embedded decoder 116. The embedded decoder 
116 comprises an inverse quantizer 122. an inverse 
DCT unit 124 a summer 126. a inter-intra switch 129. 
and a motion compensation processor 1 28. This assem- 
blage of components decodes ;he previously encoded 
block and motion compensates the block in a conven- 
tional manner Specifically, the masked block is inverse 
quantized and inverse DCT processed. As such, a block 
of reconstructed residuals is formed. These reconstruct- 
ed residuals are nearly the same as the residuals pro- 
duced at subuactor 1 04. but they lack the masked image 
information. The reconstructed residuals are summed 
with a predicted image within summer 126 and the 
summed reconstructed block is motion compensated 
within motion compensation processor 128 to form a 
predicted image. The inter-intra switch 1 30 operates in 
the same manner as switch 120 as well as also being 
under the control of the inter-intra decider 114. As such, 
the predicted image is used to form the reconstructed 
block and coupled through the switch 1 30 only when the 
residuals were originally encoded Otherwise the out- 
put of the inverse DCT unit is a reconstructed target im- 
age that is coupled to the motion compensator without 
being added to a predicted image 

[0020] It is important to note that the watermark mask 
113 is within the "loop" that encodes and decodes the 
target image to form a predicted image, while the wa- 



termark inserter 120 is outside of that loop. By placing 
the watermark mask inside the loop, an authorized 
MPEG decoder will track the embedded decoder after 
the watermark is removed 

s [0021] The watermark inserter 1 20 is outside the loop 
to ensure that the watermarked values are "branded" 
into the bitstream. Note that the watermarked DCT co- 
efficients are modified at the last possible encoding 
stage and that no further "lossy" processing occurs after 

10 the watermark coefficients "ale inserted into the masked 
array of DCT coefficients All the processes that are per- 
formed by the output processor are lossless, e.g., zig- 
zag scanning, VLC and RLC. 

[0022] Using the embodiments of watermarking ap~ 

'5 paratus according to the present invention, an unauthor- 
ized MPEG decoder will not track the embedded decod- 
er because an unauthorized decoder will not be able to 
remove the watermark from the decoded block. As such, 
the output images from an unauthorized decoder will 

20 contain periodically fluctuating noise whose visibility is 
controlled by the value of the watermarked coefficients. 
A decoder that can be used to remove a watermark em- 
bedded in the manner discussed above, is disclosed 
with respect to FIG. 5 below. 

25 [0023] FIG. 2 depicts a block diagram of the process 
used to embed the watermark into an image and FIGS. 
3, 4, and 5 graphically depict the array of DCT coeffi- 
cients as it is processed by the present embodiment. To 
best understand the process the reader should simulta- 

30 neously refer to FIGS. 2 : 3, 4. and 5. 

[0024] Specifically, the illustrative watermarking rou- 
tine 200 according to the invention begins, at step 202, 
with an input image (e.g.. the residual image). At step 
204. the input image is converted to the frequency do- 

35 mam using a discrete cosine transformation of the im- 
age content and then quantizing the DCT coefficients. 
FIG. 3 graphically depicts an 3x3 block or array 300 of 
quantized DCT coefficients The watermark mask is ap- 
plied at step 206 to select certain ones of the DCT co- 

-to efficients. At step 208, the selected coefficients are "ze- 
roed", i.e.. assigned a zero value. FIG. 4 graphically de- 
picts the block 300 of FIG. 2 after having a watermarking 
mask applied to the block and the selected DCT coeffi- 
cients 400 set to a zero value. At step 210, the routine 

^ 5 produces the masked array as an output. As described 
above, in an MPEG decoder application, the masked ar- 
ray can be used by an embedded encoder to produce a 
predicted image. 

[0025] At step 212, the routine 200 replaces the se- 
50 lectively zeroed coefficients with watermark values (i.e.. 
quantized DCT coefficients derived from a watermark 
image) Care must be taken in selecting DCT coeffi- 
cients for replacement. If the watermark is to have low 
visibility, then it is critical that relatively small watermark 
35 DCT values e g. -1 0. i 1 be inserted into the locations 
carrying the highest frequency coefficients of the DCT 
Such highest frequency locations are generally in the 
lower right hand corner of the DCT block Masking these 
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coefficients will cause a low-level "cheesecloth"-like ap- 
pearance in the reconstructed values. However, such 
an effect does not generally produce visible artifacts in 
the decoded image sequence To further reduce the vis- 
ibility of the cheesecloth effect, the watermark DCT val- 
ues that are inserted can be reversed in polarity in co- 
sited macroblocks f rom frame-to-frame. As such, the av- 
erage luminance deviation is zero and therefore less vis- 
ible. If the watermark is to have high visibility, large wa- 
termark DCT values should be used. FIG. 5 graphically 
depicts the array 300 after watermark DCT coefficients 
500 have been inserted into the masked locations in the 
array. At step 214, the routine outputs the watermarked 
array of DCT coefficients. 

[0026] The replaced DCT coefficients will affect the 
bit count associated with each block in the picture, i.e.. 
each block in a picture that is encoded is allocated a 
certain number of bits that are used to encode the block 
and the addition of the watermark values will generally 
increase that bit count. In some cases, the addition of a 
watermark, may decrease the bit count but in most cas- 
es (i.e., where the high frequency . DCT coefficients 
would have otherwise been zero values), the bit count 
will be increased. However, careful selection of the wa- 
termark coefficients can minimize the additional bit 
count. For example, it is more efficient to code coeffi- 
cients with magnitude 0 or 1 than it is to code values 
with greater magnitude. Thus, low magnitudes should 
be used as watermark coefficients. Additionally, the rate 
control technique (i.e.. the algorithm that actually con- 
trols the coding rate and quantization scale for the DCT 
coefficients) can be modified on either a local or global 
basis to account for the slight increase in bit rate that 
adding watermark coefficients will require. 
[0027] The actual watermark values may be deter- 
ministically. pseudo-randomly or cryptographically gen- 
erated within the watermark generator (1 32 of FIG. 1). 
For instance, the generator 132 converts a trademark 
(e.g.. XYZ) or some other identifying word or image into 
a binary (ASCII) equivalent. The binary values, for ex- 
ample, are then used to replace the quantized coeffi- 
cient Q(7 : 7) of FIG. 2 with a 1 or 0 in successive blocks 
that are processed by the encoder. In this manner only 
one coefficient in each array of coefficients is replaced. 
[0028] Alternatively, a pseudo-random sequence 
generator could generate a pseudo-random value, nor- 
malize the value and replace Q(7.7) with a 0 whenever 
the normalized value is in the range 0 to 0.5, and replace 
Q(7.7) with 1 whenever the normalized value is in the 
range 0.5 to 1 . The type of watermark that is used (de- 
terministic or pseudo-random) can be transmitted to the 
receiving decoder using the user data field within the 
MPEG transport stream. When using a pseudo-random- 
ly generated code, the user data can also be used to 
transmit a "seed" for the pseudo-random code such that 
the decoder can easily extract the watermark. Using the 
user data in this manner, allows the encoder that inserts 
the watermark to change the watermark code while still 



allowing the authorized decoder to automatically re- 
move the new watermark. 

[0029] Once the watermark is inserted into an image 
tn the manner described above, an unauthorized decod- 

5 er (e.g., standard MPEG) processes the watermark as 
if it were the coded image Depending upon the water- 
mark coefficients, the watermark will either be visible or 
invisible in the decoded images. In some cases, it is de- 
sirable to have a highly visible watermark that appears 

10 in any images that were decoded using an unauthorized 
decoder. However, such a visible watermark would re- 
quire removal from images decoded by an authorized 
decoder. 

[0030] FIG. 6 depicts a block diagram of a btock- 

'5 based image decoder (e.g., an MPEG decoder) for de- 
coding a bitstream generated by the encoder of FIG. 1 
and simultaneously removing a watermark from the de- 
coded image sequence. The decoder 600 comprises an 
input processor 602. a watermark remover/comparator 

20 604, and a conventional decoder 606. The input proc- 
essor 602 processes the MPEG-compliant bitstream by 
variable length decoding, run-length decoding, inverse 
zig-zag scanning, and removing the header to produce 
a block (array) of quantized DCT coefficients containing 

25 watermark information. The picture layer user data con- 
tains information regarding the manner in which the 
DCT block has been modified to include the watermark, 
i.e., the information details which locations contain wa- 
termark data. From the user data, the input processor 

30 602 produces a control signal that is coupled to the wa- 
termark remover/comparator 604. This control signal 
provides the locations of the coefficients that carry the 
watermark values. To verify the authenticity of the wa- 
termark, the watermark remover/comparator extracts 

35 the watermark values from the block and the extracted 
watermark coefficients are compared against a stored 
reference (or a reference that is pseudo-randomly gen- 
erated from a seed transmitted from the encoder to the 
decoder). If a match occurs, the authenticity of the wa- 

40 termark is confirmed and an authentication signal is pro- 
duced. If no match is found, then the watermark was not 
inserted by an authorized encoder. If a match occurs, 
the watermark remover/comparator then "zeroes" the 
watermark value in each of those locations to produce 

J 5 the block Z (also depicted in FIG. 4). However, without 
a match, the watermark remover/comparator will not ze- 
ro the watermark values and. as such, any reconstruct- 
ed images will be distorted by the watermark values. 
[0031] To complete decoding of the bitstream, the 

50 MPEG decoder 606 comprises an inverse quantizer 

605 an inverse DCT unit 610. a summer 612. a inter- 
intra switch 614 and a motion compensator 616. The 
motion vectors and the intra/inter decisions associated 
with each block are extracted from the bitstream by the 

55 input processor 602 and coupled to the MPEG decoder 

606 The inverse quantizer 606 and inverse DCT unit 
610 operate upon block Z to inverse quantize and in- 
verse DCT process the block to form a block of pixels 
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The summer 612 sums the pixels with a predicted block 
of pixels to form a reconstructed image, tf the block was 
intra-coded in the encoder then a predicted block is not 
applied to the summer 612 i.e.. the switch 614 is 
switched to the intra terminal which is grounded. The s 
reconstructed image is produced at the decoder output 
and also coupled to the motion compensator to be mo- 
tion compensated to form a predicted block. 
[0032] Although various embodiments which incorpo- 
rate the teachings of the present invention have been w 
shown and described in detail herein., those skilled in 
the art can readily devise many other varied embodi- 
ments that still incorporate these teachings. 

15 

Claims 

1 . An image encoder for inserting a watermark into an 
image containing a plurality of pixels, where the pix- 
els are encoded using discrete cosine transform 20 
(DCT) coefficients, comprising: 

a watermark mask for masking an array of DCT 
coefficients representing said image and set- 
ting select ones of said DCT coefficients in said 25 
array to a zero value: and 
a watermark inserter coupled to said water- 
mark mask, for inserting watermark values into 
locations in said DCT coefficient array where 
said DCT coefficients were masked and set to 30 
a zero value. 

2. A DCT based image encoder for inserting a water- 
mark into an image containing a plurality ofpixels. 
where the pixels are encoded using discrete cosine ^5 
transform (DCT) coefficients, comprising: 

a watermark mask for selecting certain ones of 
said DCT coefficients representing said image 
and changing the value of the selected DCT co- -to 
efficients to zero to form a masked array of DCT 
coefficients: 

an image decoder, coupled to said watermark 
mask, for generating a predicted image from 
said masked array of DCT coefficients: -ts 
a watermark inserter, coupled to said water- 
mark mask, for replacing the selected DCT co- 
efficients with watermark values to produce a 
watermarked array of DCT coefficients: 
an output processor, coupled to said watermark 50 
inserter, for losslessly coding said watermarked 
array of DCT coefficients to form a bitstream. 

3. The apparatus of claim 1 or 2 further comprising a 
watermark generator for producing said watermark 55 
coefficients as quantized DCT coefficients of a pre- 
defined character string. 



4. The apparatus of claim 1 or 2 further comprising a 
watermark generator for producing said watermark 
values are pseudo-randomly generated. 

5. The apparatus of claim 4 wherein said watermark 
generator is coupled to said output processor and 
supplies said output processor with certain informa- 
tion useful in decoding said watermark values, 
where the output processor inserts the certain infor- 
mation into said bitstream. 

6. The apparatus of claim 5 wherein said certain infor- 
mation is a seed for decoding said pseudo-random- 
ly generated watermark values. 

I. A method of inserting a watermark into an image 
containing a plurality of pixels, where the pixels are 
encoded using discrete cosine'transform (DCT) co- 
efficients, comprising the steps of: 

masking an array of DCT coefficients repre- 
senting said image and setting select ones of 
said DCT coefficients in said array to a zero val- 
ue: and 

inserting watermark values into locations in 
said DCT coefficient array where said DCT co- 
efficients were masked and set to a zero value. 

8. A method of extracting a watermark from a bit- 
stream comprising the steps of: 

extracting, from the bitstream, DCT coefficients 
that contain watermark values: 
comparing the watermark values to reference 
watermark values: 

producing an authentication signal if the water- 
mark values match the reference watermark 
values: 

if said authentication signal is produced, setting 
said watermark values in said DCT coefficients 
of said bitstream to zero to remove the water- 
mark from the DCT coefficients in the bitstream, 
whereby an image is decoded without the ap- 
pearance of a watermark. 

9. The method of claim 8 further comprising the step 
of recalling said reference watermark values from 
memory. 

10. The method of claim 8 further comprising the step 
of 

decoding information within said bitstream to 
produce said reference watermark values. 

I I . The method of claim 1 0 wherein said information is 
a seed for a pseudorandom code that forms the ba- 
sis for the reference watermark values 
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(54) Video signal processing for electronic watermarking 



(57) An image or sequence of images is water- 
marked without limiting the watermark signal. The wa- 
termarking apparatus includes a conventional DCT unit 
(103) and quantizer (110) for generating an array of 
quantized DCT coefficients. The array is watermarked 
(113) by masking the array to select certain ones of the 
DCT coefficients that are then replaced by zero values 



to form a masked array. The masked array is further 
processed by a watermark inserter(1 20) that replaces 
the zero valued coefficients with predefined watermark 
coefficients to form a watermarked array of DCT coeffi- 
cients: e.g., a watermarked image. A decoder for decod- 
ing the bitstream thusly generated and for removing the 
embedded watermark is also taught. 
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Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 
El FADED TEXT OR DRAWING 

□fBLURRED OR ILLE GIBLE TEXT O JRDRAWING— 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



